1. Field of the Invention
The present invention relates to a method and an apparatus that enables a second user to share a mixed reality space image including a virtual object superimposed in a space where a first user exists.
2. Description of the Related Art
A seamless mixing of a real world and a virtual world can be realized based on a recently developed mixed reality (MR) technique. The MR technique enhances a VR (virtual reality) technique so that the coexistence of a physical space and a virtual reality world can be realized, whereas, in the past, the VR world can be experienced only in a situation separated from the physical space.
A head mounted display apparatus (HMD) is a representative device required to realize the mixed reality. More specifically, the HMD can display a combined image of a physical space and a virtual space.
In a VR-based system for supporting a conference or another cooperative work performed among remote participants, a computer creates an above-described virtual world independent of the real world to let the remote participants share the created virtual world. On the other hand, an MR-based remote supporting system enables remote users to support the work of another user in a work space including a virtual world superimposed in the real world.
According to conventional techniques, remote participants can share a VR space and can observe the space by changing their viewpoints. For example, in a game machine, virtual spaces displayed for individual players can be combined so that other players can flexibly display the composite space from a desired angle and a desired position. The space created by a game machine is a virtual space that is independent of the real space. The game machine cannot realize a mixed reality including a virtual object superimposed with the object(s) in the real world. It is thus impossible to observe both the real object(s) and the virtual object(s) simultaneously by changing the viewpoint.
As discussed in Japanese Patent Application Laid-open No. 2002-132487 (corresponding to U.S. 2002/0049510 A1), a head mounted display apparatus (HMD) can be used to support remote working. According to the example, a camera-equipped HMD is mounted on a worker and both the worker and a remote instructor can share the image of the camera. The instructor can give a work instruction by superimposing the instruction on the camera image.
When the worker's camera is a stereo type, both the worker and the instructor can observe the same stereoscopic image. Furthermore, based on a chromakey combination method, an instructor's hand pointing to a work object in the image can be combined with the camera image.
Furthermore, according to the above-described conventional technique, a stationary camera is placed in addition to the worker's HMD and the image of the stationary camera can be transmitted to a remote instructor to let the instructor support the work. However, this system does not use the coordinate system defining the worker's space. Thus, it is impossible to set a virtual object other than the work object at an arbitrary position of the worker space to perform pointing or other interactions. Furthermore, this system does not use plural cameras placed in the worker's space and accordingly cannot allow a remote instructor to observe from plural viewpoints.
As discussed in Japanese Patent Application Laid-open No. 2001-195601 (corresponding to EP 1117074), plural objective viewpoint cameras can be fixed in a mixed reality space. A composite image including a virtual object superimposed on an image of this camera can be selectively displayed on plural display devices. However, this system cannot allow a remote instructor to perform an interaction, such as pointing or manipulating a virtual object, in the mixed reality space while observing the displayed image.